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ABSTRACT 

Generalizability theory was used to analyze the 
dependability of elementary school student ratings of attitudes 
toward school subjects. The rating scales under investigation have 
teen developed tc measure the attitudes of students toward four 
school subjects at both the primary and intermediate levels- Two 
generalizability coef f icients , differing in assumptions of true and 
error variance, were calculated which estimated the dependability of 
assessing class attitude* A third coefficient was calculated which 
estimated the dependability of assessing individual student 
attitudes. The effects of varying the rumber of scale items and/or 
students on the dependability of class means or individual student 
attitudes were investigated. The results suggest the attitude scales 
measuring primary student attitudes toward school subjects are not 
dependable indicators of student attitudes when class is the unit of 
analysis. There is cogent evidence that both primary and intermediate 
student attitudes can be dependably evaluated by the attitude scales 
when the student is the unit cf analysis. (Author) 
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Abstract 

Generalizability theory was used to analyze the depend- 
ability of elementary school student ratings of attitudes 
toward school subjects. The rating scales under investigation 
have been developed to measure the attitudes of students 
toward four school subjects at both the primary and inter- 
mediate levels. Two generalizability coefficients, differing 
in assumptions of true and error variance, were calculated 
which estimated the dependability of assessing class attitude. 
A third coefficient was calculated which estimated the depend- 
ability of assessing individual student attitudes. The 
effects of varying the number of scale items and/or students 
on the dependability of class means or individual student 
attitudes were investigated. The results suggest the attitude 
scales measuring primary student attitudes toward school sub- 
jects are not dependable indicators of student attitudes when 
class is the unit of analysis. There is cogent evidence that 
both primary and intermediate student attitudes can be depend- 
ably evaluated by the attitude scales when the student is the 
unit of analysis. 
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The Generaiizability of Elementary School 
Student Ratings of Attitudes Toward 
Schoo 1 Subj ects 



Rating scales designed to measure elementary school 
students' attitudes toward school subjects can provide educa- 
tors with essential information to assist them in identifying 
particular students' attitudes, planning instructional pro- 
grams, determining a class's general attitude toward a 
curricular area, curriculum evaluation and school attitude 
research (Hogan, 1975, pp. 4-5). The dependability (reliabil- 
ity) of attitude instruments is an important concern when 
making decisions of this nature. Factors such as the number 
of students taking the attitude instrument and the number of 
items included on the instrument have a pronounced effect on 
the instrument's dependability for assessing class attitude. 
The number of items included on the instrument has a pr onounced 
etfect on the dependability of assessing individual student 
attitudes. The manner in which the dependability coefficient 
is defined is also an important consideration. 

Previous research (Gillmore, Kane, and Naccarjto, 1978; 
Kane, Crooks, and Gillmore, 1976; Douglass, Note 1) has 
demonstrated the usefulness of generaiizability theory (Cron- 
bach, Gleser, Nanda, and Rajaratnam, 1972) in analyzing the 
dependability of student ratings of instruction. These studies 
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have also addressed a number of concerns in the definition of 
dependability coefficients 

The present study extends the application of generaliz- 
ability theory to the analysis of the dependability of 
elementary school student ratings of attitudes toward schooL 
subjects. The rating scales under investigation have been 
developed to measure the attitudes of students toward four 
school subjects at both the primary (grades 1, 2, and 3) and 
intermediate (grades 4, 5, 6, 7, and 8) levels. The effects 
of varying the number of scale items and students on the 
dependability of individual score and classroom means were 
assessed for each of the scales at the primary and inter- 
mediate levels. 

The studies previously cited on student ratings of in- 
struction have considered the class as the unit of analysis. 
Similarly, in the present investigation the mean rating of a 
class was considered as the object of measurement in some of 
the analyses. Other analyses treated the student as the 
object of measurement since individual student attitude was 
also considered to be important in the present situation. 

The first step in completing a generalizability theory 
study is to estimate the components of variation from the 
sample data. This is referred to as the generalizability (G) 
study. The components of variation can then be arranged to 
form generalizability coefficients which reflect the depend- 
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ability of measurement in any of a variety contexts. This 
second stage is referred to as the decision (D) study. 
General inability coefficients are defined as the ratio of the 
universe score variance to the expected observed score variance. 
They have the general form of a reliability coefficient in 
classical test theory if the universe score variance is 
defined to be equal to the true score variance. Generaliz- 
ability coefficients differ according to the definition of 
the sources of variation constituting universe and expected 
observed score variance, and the number of sampling units 
included. 

In the present study, students were nested within classes 
and each student was measured with the same instrument. Hence, 
the components of variation were estimated using a split-plot 
design with classes, students and items considered to be 

random effects. 

The estimated components of variation considered for this 

investigation were, 

student,cr 2 OT), which is an estimate of the true variance 
of students nested within classes; 
class, <r 2 (<*), which is an estimate of true class 
variance ; 

item, <r 2 (/9), which is estimate of the true item 
variance ; 

Class x item, a- 2 (^), which is an estimate of the class 
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by |. Lorn interaction vairi.atl.oii; 

and cesiduaL, <r £ 2 , which La an estimate ot: the •u:i:ol- 
variance . 

The generaiizabiiity coefficients calculated i.n tlx La 
study are presented be Low. 

One generaiizabiiity coefficient calculated to estimate 
the dependability of class means was: 

p2(s,D- . Z 2 fr) 



[ff 2 (oc)]+ [g- 2 (7T)/n + fr 2 ^/9)/k + f e 2 /nk] , (D 

where, n = number of students in the class, and 
k = number of items. 
This general reliability generaiizabiiity coefficient 
considers students and items as sources of error, and it 
represents the expected correlation between two means for 
each class where the means are based on a different sft of 

items and students. 

The second generaiizabiiity coefficient calculated to 
estimate the dependability of classroom means was: 

p 2 (s,i*) = frV) t fr 2 fr*>/i£ - 

[a 2 (°<) +O i2 (^)/k]+[o i2 (Tr)/n+^ 2 /nk] . (2) 

This student reliability generaiizab iiity coefficient 
considers students as a source of error, and it represents 
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r.his nspiM 1 \ iu\ ^orrnLil Ion hat.wmui t wo moan.H i: * » i* i^u'.h ' l.i.^i 
wlniiru Uhu muauH art* ha.suil mi a ilMiCiU'enL -uit; a I: stauU.MiL.;; on 
Clio rUiinii LtUim.s. Tho ikn:LvaU.onH o! : aqua I: I.oum I. and J! aro 
(llruaiHsuil lai Kanu and iUrminan ( L0 77 f pp. 2/l-l/H). 

In Dixit* r t:o osLlmat.o. Lho il op inula h I. U.ty ot; stauUait; ricoros 
within a cLaaa the La L Lowing w L U h L n- c La a h g u no ca L L za b 1 1. 1. L y 
coof £ Lciont wa;j calculated; 

P 2 £ 2 (TT) 

V w-c »- z - 0 ,^ 

CT- 2 CTT) + ? 2 /k • < 3 > 

This coefficient represents the correlation between 
scores of different sets ot items for students within a given 
class. It would be expected to be approximately equal to the 
average within-class coefficient alpha value. 

Method 

Sample 

The sample consisted of 160 first, second, third, fourth, 
fifth, and sixth grade students belonging to a school district 
located near the New York Metropolitan area. Eight primary 
and eight intermediate classes, each containing ten students, 
were included in the analyses. Students were randomly selected 
from classes containing more than ten students. 
Design and Procedure 

The design used for this study was a split-plot design 
(Winer, 1971) and designated as design V-B by Cronbach et. al. 
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I- ii whU'U atudentH are nuMtad wU.ULu i- lassos ami 
cro.saed with Items. Each student responded each Item in 
all four scales and the variance components were cat .Una laid 
separately for uach Lical.i allowing for eompnr I .sons anion thu 
reliability coeffllc Lents of the I: our scales. Additionally, 
an analyaLs was completed for the sum of the tour scales. 
A separate analysis was performed for the primary Level form 
and the intermediate level form and the resulting generallz- 
abllity coefficients were compared across levels. 
Ins trumentation 

The rating scale used to measure attitudes towards 
school subjects was the primary (grades 1, 2, and 3) and 
intermediate (grades 4, 5, 6, 7, and 8) form A of the Survey 
of School Att : '-'.des by Thomas P. Hogan, 1975 Edition, pub- 
lished by Harcourt Brace Jovanovich, Inc., New York. This 
survey consists of four scales of 15 items each in the areas 
of Reading/Language, Mathematics, Science, and Social Studies. 
All items are three- point Likert-type items. 

Results 

The estimated variance components for the primary and 
intermediate scales are presented in Table 1. Following the 
suggestion of Kane and Brennan (1977, p. 290), negative 
estimated variance components have been replaced by zeros. 



Insert Table 1 about here 
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i'lxc.ipl: l:oi: the primary uiuthtmiat Los sraLo, the r<;iual ni.11.4 
primary MciLea luul an eMtl.uiat.ul elans variance, q- ,! (,«), equal, 
to zero, The value of the out Una I. ml cLass varLaneu was a Lao 
equal, to zero I: or the .-mm ot: all. ft mi: primary scales, ami l.lus 
estimated varLance for the cLass by Item Lnteraetion, 
was equal to zeiro I: or the primary social studies scal.e. The 
estimated student variance, fr 2 (Tr), wa:i greater than the esti- 
mated item variance, <r 2 (j9), for each of the primary and 
intermediate scales. 

Generalizability coefficients estimating the dependability 
of class means are presented in Table 2. Coefficients are 
tabulated for samples of five and ten students and five and 
fifteen items. This was done to indicate the effects of vary- 
ing the number of scale items and students on the dependabil- 
ity of class means. 



Insert Table 2 about here 
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Four three of the four primary scales and the sum of 
scales, the general reliability was zero. The exception was 
the primary mathematics scale since it had > nonzero esti- 
mated class variance. 

Except for the primary mathematics and social studies 
scales, the student reliability tended to decrease with in- 
creasing numb- of items for the remaining primary scales. 



This L'a.suLUul h'<mi >ui usl Uiat o»l .lass varum*. *> u.|iutl. i-i /urn 

i.u the muiiauiUnr >>l the -lUuluu. r«^UhjJ_U^ l it al) )- U.. 1 

.'.iu! if if Lc L ent . 

Kou both t-lut inl.«u.(tUMlial ♦! .111.1 [.riuuu'V Staler,, \\u-. v>\u>\\\\ 
reliability coefficient was always nnui L I.u 1: than I: he corre- 
sponding student reliabLLLty coot: lLcU.mil because U contained 
onLy the estimated class vat'Lauce In the estimated universe 
score variance. In particular, for small numbers of Items, 
the student reliability coefficient was considerably Larger 
than the corresponding general reliability coefficient fur thi 
intermediate reading/ language and science scales and the pri- 
mary math scale. 

Increasing the number of students tended to have a 
greater effect on the magnitudes of the nonzero generaliz- 
ability coefficients than increasing the number of items, for 
both the primary and intermediate scales. Furthermore, large 
increases in items had a relatively small effect on the magni 
tude of generalizability coefficients. For example, the 
student reliability coefficient for the intermediate reading 
scale which was .61 when n = 10 and k - 15, increased to only 
.64 when k = 60 and n was held constant. 

Estimated within-class eeneralizabilit y coefficients 
are presented in Table 3 for samples of five, fifteen, and 
sixty items. 
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i'pi lor ^ ii-nr.i!, fhti priuiarv ualu w i I. hi n-»- la^.i 
^iiti iviLi/ahi M t y I'-oar" rMiM-eui^ wtu-u ^i:ua(.^r than the iatui ■ 
ituuli.atu si^iLcj I'.Diitt LctouU.-j far al l numbers at Ll-uina, The 
i.t s 1 1 ma Led wj LULu- c Uhm ^auu ral l./abl 1. 1 1 y coeLt: Lc Lei U, c a L c u - 
LaLoil Lu Lhl-H study l:m: mmpLeH or l: Lr: teeeu l.Luiua aro present* 
Lu Table t* wLLh the eorretfpomlLng Ooef fix Lout Alpha e.stl.iuatua 
h< f calculated by llo^au (L l )7 r ) [a], p.Mj LW [h] f p. 8), 
The Coeft: LoLent ALpha estLuiates were eaLeuLated from the 
responses of over 2,000 primary and 4,000 LntermodLate stu- 
dents. Of speelaL note Ls the generaLLy cLose agreement 
between the corresponding estimated wL thin-class and ALpha 
Coefficients. The coefficients for the primary scales agreed 
much more ciosely then did the intermediate scaie coefficient 
however, this may have been partially due to the absence of 
grade 7 and 8 students from the present study. 

Discussion 

Of the three generalizability coefficients presented 
in this study the student reliability generalizability co- 
efficient is perhaps the least useful because generalization 
is usually desired beyond the fixed set of items included 
on the instrument. The general reliability generalizability 
coefficient .is useful when the class is the unit of analyses 
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and generalization is desired beyond the finite number of stu- 
dents and iterajj included in the study. The within-class 
g eneralizability coefficient is a useful estimate of the 
dependability of scores when the student is the unit of 
analysis . 

The results of the present study indicate that the 
estimated item variance is consistently less than the esti- 
mated student variance. This result is in agreement with 
studies that have been done on student ratings of instruction 
(GUlmore et. al., 1978, p. 1). Thus, increasing the number 
of items generally has a lesser effect on the dependability 
of class means than increasing the number of students. 

There was a tendency for the primary scales to have 
estimated class variance at or near zero which was reflected 
in the low general and student reliab i lity generalizability 
coefficients . This result suggests that in the primary grades 
the class or teacher has relatively little effect on student 
attitudes toward school subjects. Thus, the attitude 
scales measuring primary student attitudes toward school 
subjects are not dependable indicators of student attitudes 
when the class is the unit of analysis. However, there is 
cogent evidence that both primary and intermediate student 
attitudes can be dependably evaluated by attitude scales when 
the student is the unit of analysis. 
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The present investigation might have been improved by 
using an attitude scale containing more than three response 
options per item. This would probably result in relatively 
larger estimated variance components. Results might also 
have been more consistent had larger classes been used, and 
had grades 7 and 3 been included in the analyses. 

The SSA (Hogan, 1975) uses three faces (a smiling, 
neutral, and frowning face) for response options for each 
item. It would be interesting to complete a study on the 
effect of this kind of response option compared to other types 
of response options on the dependability of elementary student 
responses . 
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Estimated 

variance 

component 



Table 1 

Estimated Variance Components for Primary and 
Intermediate Scales 



Reading/ 



Social Sum of 



language Mathematics Science studies scales 



Primary 



n ace 

via o o 


0000 


021 8 

• \J <t» X U 


noon 


oooo 


0000 

l vvvv 


Students 












within class 


. 1034 


. 1505 


.0977 


. 1437 


.0956 


Items 


.0283 


.0303 


.0112 


.0272 


.0301 


Class x items 


.0156 


.0215 


.0297 


.0000 


.0183 


Residual 


.4364 


.4291 


.4489 


.4435 


.4677 


Intermediate 












Class 


.0172 


.0042 


.0101 


.0178 


.0102 


Students 












within class 


.0886 


. 1119 


.1173 


.0989 


.0604 


Items 


.0610 


.0252 


.0211 


.0346 


.0370 


Class x items 


.0228 


.0055 


.0124 


.0056 


.0138 


Residual 


.4339 


.4265 


.3636 


.3906 


.4474 
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Scale 



Table 2 

Estimated Generalizability Coefficients for 
Reliability of Class Means 

Items per scale 

_ 5 . 15. 

Number of students 
5 10 5 

Primary scales 



General reliability 
Reading/ language 
Mathematics 
Science 

Social studies 
Sum of scales 
Student reliability 
Reading/ language 
Mathematics 
Science 

Social studies 
Sum of scales 



.00 
.30 
.00 
.00 
.00 

.08 
.36 
.14 
.00 
.04 



.00 
.44 
.00 
.00 
.00 

.14 
.53 
.24 
.00 
.07 



.00 
.37 
.00 
.00 
.00 

.04 
.39 
.07 
.00 
.02 



10 



.00 
.53 
.00 
.00 
.00 

.07 
.57 
.13 
.00 
.03 
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Scale 



Table 2 (continued) 
Estimated Generalizability Coefficients for 
Reliability of Class Means 

Items per scale 

5 15 
Number of students 

5 10 5 10 
Intermediate scales 

General reliability 

Reading/language .30 .44 .41 .57 

Mathematics .09 .17 .13 .23 

bcience .20 .32 .26 .40 

Social studies .33 .49 .41 .58 

Sum of scales .37 .53 .42 .59 

Student reliability 

Reading/language .38 .56 .44 .61 

Mathematics .12 .21 .14 .25 

Science .25 .40 .28 .44 

Social studies .35 .52 .42 .59 

Sum of scales .40 .57 .43 .61 
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Scales 



Table 3 

Estimated Within-Class Generalizability Coefficients 
for Primary and Intermediate Sclaes 

Items per scale 
5 15 



Primary 

Reading/language 

Mathematics 

Science 

Social studies 
Sum of scales 
Intermediate 

Reading/language 

Mathematics 

Science 

Social studies 
Sum of scales 



54 
64 
52 
62 
80 

,51 
,57 
,62 
,56 
,73 



78 
84 
77 
83 
92 

,75 
,80 
,83 
,79 
,89 



93 

,95 
,93 
,95 
,98 

,92 
,94 
,95 
,94 
.97 
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Table 4 

Comparison of the Within-Class Coefficient in the 
Present Study With the Results of Hogan 
(1975 fa] , p. 3; 1975 fbj , p. 8) 



Subject 



Primary scales 

To*, calculated 
w-c by Hogan 



Intermediate scales 



ft 



calculated 



by Hogan 



Reading/ language 

Mathematics 

Science 

Social studies 



78 
84 
77 
83 



.81 
.85 
.79 
.84 



75 
80 
83 
79 



.84 
.92 
.84 
.84 
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